Repository navigation
offline profile 5: the conversation data export streams from the server when no S3 bucket is configured (stacked on #2932) - #2940
Open
colinmegill wants to merge 16 commits into
Open
colinmegill wants to merge 16 commits into
colinmegill wants to merge 16 commits into
Conversation
One file names the vote sign. It owns the wire convention (frozen at
agree = -1; named WIRE_AGREE/WIRE_DISAGREE/WIRE_PASS), the export
convention (fixed at agree = +1), and the storage convention, read from
an injectable ConventionSource whose only implementation today is the
constant STORAGE_AGREE_VALUE = -1. The pure conversions map between the
three numeric forms and a typed Vote ('agree' | 'disagree' | 'pass').
NULL in is NULL out for every conversion; the CSV exports' NULL cell is
the named EXPORT_NULL_VALUE (0, as today). Out-of-range integers throw
unless the caller opts into "skip" (semantic reads) or "keep" (keep
today's bytes), so a legacy tolerance is visible at the call site.
Unit tests cover every function, NULL, out-of-range values, round trips
under both storage conventions, and an injected source.
No call site changes in this commit. No schema change. No API change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
report.ts had three hand-written flips (votes.csv String(-row.vote), participant-votes.csv and participant-importance.csv -row.vote) and two opposite literal tables in one file: comments.csv counted -1 as agree, participant-votes.csv counted 1 as agree, each right only because of where the flip sat. Both now classify the stored value once through storageToSemantic and count by name; the cells are written by storageToExport (agree = +1). The importance export never used the flipped value, so its map is gone and n-votes counts the voted tids. The bytes do not change: a new test pins every vote column of the four exports for agree, disagree, pass, NULL (written 0, as before) and an out-of-range stored value (written negated, as before), and passes on both the old and the new report.ts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The import swapped exact +1/-1 by hand ("INTENTIONAL VOTE FLIPPING,
REMOVE AFTER VOTES REFACTOR"). vote_value is in the export convention
(agree = +1, as the admin import screen documents), so the reader now
converts it with exportToStorage. Values other than -1/0/1 are still
stored as given ({ onInvalid: "keep" }); a new test pins that, and it
passes on the old and new code. Refusing them belongs to the declared
import format change, not here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
POST /votes and POST /comments parse the vote with getIntInRange(WIRE_VOTE_MIN, WIRE_VOTE_MAX) instead of a bare (-1, 1). doVotesPost, the single INSERT reached from votes, comments and the bulk seed upload, converts the wire value with wireToStorage before writing; today wire and storage agree (-1), so the stored value is unchanged, and when the storage convention changes only the source changes. The seed comments' default author vote is WIRE_PASS rather than a bare 0. The (unreached) modOptions branch reads the vote by name rather than through the push/pull reaction literals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
- GET /votes, GET /votes/me and participationInit's votes return each row through storageRowToWire (wire = storage today, so the value and the JSON key order are unchanged; an out-of-range stored value is returned as stored, as before). - The moderation list counts (comment.ts) and the famous/ptptoi letter vectors (server-helpers.ts) classify the stored value by name through storageToSemantic instead of the push/pull reaction literals; NULL and unknown values keep their old treatment (counted nowhere, left 'u'). - The collective statement query builds its CASE predicates from storageSqlValue; the SQL text is the same as before. - polisTypes.reactions (push/pull/see/pass), which never said agree or disagree, is retired; nothing reads it any more. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
A unit test reads every server source file (src/, app.ts, index.ts, bin/, scripts/) except src/votes/convention.ts and fails on a line that names a vote and carries a sign literal or a negation: -1, === 1, * -1, -row.vote, vote < 0, vote = 1 in SQL, or the retired push/pull reaction names. It also checks that the patterns catch the shapes the audit found (and do not catch group-1 or an index sentinel), and that every allowlist entry still matches, so the list can only shrink. After the routing commits the scan finds nothing and the allowlist is empty. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The witness executes the real votesGet / handle_GET_votes_me source in a vm context. Those routes now call storageRowToWire, so the witness transpiles src/votes/convention.ts as-is and exposes its exports in the same context (nothing is copied). A source tree that predates the module gets none, since its routes do not reference it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ion module The wire witness now runs votes.ts together with the real src/votes/convention.ts, so the test's copied source tree includes it, and mutation A targets the routed read (storageRowToWire) instead of the old bare resolve(results.rows). With a local passfile DSN all eight wire tests pass; in CI they fail the same way before and after this change (the fixture's password DSN is refused by the DSN admission check). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…format.json) The CSV exports use agree = 1, disagree = -1, pass = 0, but no file said so. The convention module now builds one declaration from its export constants, agree=+1;disagree=-1;pass=0;format=polis-export/1 and the exports carry it without changing a CSV header or cell: - summary.csv gains one last key/value row, vote-convention,<declaration>. Every row above it keeps its position and bytes. - /api/v3/reportExport/:report_id/format.json is a new sidecar beside the CSVs: the format id, the declaration, the vote values, and per file which columns carry a vote value, a count, or an importance flag. The module also reads a declaration back (the declaration string or a format.json document) and refuses one that is malformed or that names another convention, with two closed codes; the votes-bulk import uses that next. The byte pins gain the summary.csv and format.json goldens (fixture files under __tests__/fixtures/vote-declaration), and assert that summary.csv minus its last row is the pre-declaration bytes. The vote CSVs' pins are untouched. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
A votes-bulk CSV may now declare its sign, on a first line before the header, # vote-convention: agree=+1;disagree=-1;pass=0;format=polis-export/1 or as a format field in the request body with format.json's shape. - Absent: the file is read in the export convention, exactly as before (the route stores the same bytes; the worker reads the same rows). - Present and the export convention: accepted. The worker drops the declaration line before the CSV header is parsed and converts every value through exportToStorage, as for an undeclared file. - Present and another sign or format: refused with polis_err_vote_convention_declaration_mismatch; unreadable: polis_err_vote_convention_declaration_malformed. The route answers 400 before anything is stored or enqueued; the worker checks the stored file's first line again and fails the job with the same code before opening a transaction. Any other first line, including another # line, still reaches the CSV parser as the header. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
docs/export-format.md is the in-repo statement of the sign of every export file and column (until now only the off-repo export page and the admin import screen said it): the export convention, the summary.csv row and format.json sidecar that declare it, each file's vote-valued and count columns, and the votes-bulk import rule with its two closed error codes. Linked from docs/configuration.md, delphi/docs/QUICK_START.md, the report page's raw data export section (which also links format.json and lists its curl line) and the admin vote import screen (which mentions the optional first-line declaration). A unit test reads the convention module's declaration cases from a fixture file and checks that the page states the declaration, the closed codes, and every file and column format.json lists. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…lumns The empty-math integration test compares loadConversationSummary with rows recorded on edge. It now asserts that the last row is the vote-convention declaration and compares the rows before it with the golden, which is not re-recorded on this branch. format.json and docs/export-format.md now also list the two importance columns, neither of them a vote value: votes.csv important (a flag) and comments.csv importance (a count). Fixes the test file name in exportFormat.ts's header. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…en no S3 bucket is configured GET /api/v3/dataExport queued a generate_export_data task for the Clojure export worker, which has not consumed worker_tasks since 2022, and its results route signs a URL into the AWS polis-datadump bucket. With AWS_S3_BUCKET_NAME unset, or OFFLINE set, the route now builds the zip on request instead: format.json, summary.csv, comments.csv, votes.csv and participant-votes.csv from the report export functions, streamed entry by entry (deflate in 256 KiB sync-flushed runs, data descriptors, no zip64). A conversation over DATA_EXPORT_MAX_CELLS (max of vote rows and voters x comments; default 20,000,000) is refused with 413 polis_err_data_export_too_large before any byte. The results route answers 404 polis_err_data_export_results_not_stored in that mode. The ownership gate is unchanged. The OFFLINE config key is #2937's definition, verbatim, so the two merge cleanly. With a bucket set and OFFLINE unset nothing changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…o the report export Generated fixture conversation written straight to Postgres; both handlers mounted on a bare Express app; every zip entry compared with the bytes /api/v3/reportExport serves. Also the OFFLINE-with-bucket mode, the 413 refusal and the unchanged ownership refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…knob Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… its reads to the client Review of #2940 (F-1): a client that disconnected left the handler pending forever (a destroyed response never emits finish), and the producers read and compressed the rest of the conversation into a dead socket. - src/db/streamControl.ts: an AsyncLocalStorage context for the Postgres row streams. stream_queryP_readOnly, inside it, destroys the running stream on abort (cursor closed, client discarded), fails a stream that has not started, and pauses while the output needs draining. Outside a context nothing changes. - writeConversationZip aborts on the response's close and runs each producer inside the context with the response as the output. - ZipStreamWriter: writes to a destroyed or ended output throw polis_err_zip_output_closed; finish() resolves on finish and rejects on close. Each entry also carries the UTC extended timestamp (0x5455). - The handler logs a client close at info and returns. - The size count reads votes once (COUNT and COUNT DISTINCT together). - F-2: positiveIntOr no longer sits under isTrueOrBlank's comment. Tests: unit for the destroyed output, the abort reaching the producer and the handler returning; unit for the stream control (abort mid-stream, abort before start, pause and resume). Integration on a 600,000-vote generated fixture: a client that stops reading holds the votes.csv query paused, and on disconnect the handler returns and the backend is gone; a reading client gets every row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Contributor
Delphi Coverage Report
|
colinmegill
added a commit
that referenced
this pull request
Oct 3, 2026
…ocal narratives example.offline.env sets OLLAMA_HOST=http://ollama:11434: the server reads it for the narrative and collective statement and would otherwise use localhost. docs/offline.md takes the lines the data export PR (#2940) and the Ollama narrative PR (#2945) left for it: checklist item 6 adds the conversation data export zip, item 7 has the narrative and collective statement from the local model, and the known gaps drop the Anthropic-only line and describe the streamed export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2932 (
p078-e-export-import-declared-sign). The first five-plus commits in this PR's diff againstedgeare #2932's. This PR adds four commits on top:56ab1abe7,66f2ae876,8454ba205,f4e9269a6(review follow-up). Merge #2932 first, or review only those three.Offline profile item 10: the conversation data export works without S3.
Decision
GET /api/v3/dataExporthas no producer, so when no bucket is configured the server streams the zip itself.Evidence:
generate_export_datarow intoworker_tasks(server/src/utils/common.tsdoAddDataExportTask) and answers{}. The consumer was the Clojure math worker's export task. Production has not consumedworker_taskssince 2022-04-18: the task system was commented out of the worker's full system in7d6db8eb6("fixes math worker crashes when run in dev mode #1279"). client-admin has no caller./api/v3/dataExport/resultssigns a URL into the hard-coded AWS bucketpolis-datadump, with no endpoint override (server/src/routes/dataExport.ts:6-10onedge)./api/v3/reportExport/<report_id>/<file>,server/src/routes/export.ts) read Postgres and stream their CSVs directly, so they need no S3. They are the producer that already works offline.What changes:
AWS_S3_BUCKET_NAMEunset, orOFFLINEset (1/true): after the existing ownership check,GET /api/v3/dataExportstreamspolis-export-<conversation_id>-<millis>.zipin its response. The zip holdsformat.json,summary.csv,comments.csv,votes.csvandparticipant-votes.csv. Each one is produced by the same report-export function and is byte-identical to that route's output. The writer holds at most one 256 KiB run of input: entries are deflated in sync-flushed runs as the rows stream out of Postgres, and data descriptors carry the CRC and sizes. The Postgres streams pause while the response needs draining, so a slow client slows the reads instead of growing Node's socket buffer. A client that disconnects aborts the export: the running query's cursor is closed, its client is discarded and the handler returns. This works through an AsyncLocalStorage context (src/db/streamControl.ts) thatstream_queryP_readOnlyreads, and it changes nothing outside that context.comments.csvandsummary.csvare each still built as one string, as in the report export. There are no zip64 records.GET /api/v3/dataExport/resultsanswers 404polis_err_data_export_results_not_stored, because no dump is stored and a signed URL would point the browser at AWS.DATA_EXPORT_MAX_CELLS, it answers 413polis_err_data_export_too_largewithcellsandlimit.hybridAuthplusisModerator). The report export is keyed by an unguessablereport_id, whereas this route takes aconversation_id, which is public. Copying the report export's auth here would therefore remove the owner check, so this PR does not.OFFLINEunset: unchanged. The route still queues the worker task and answers{}, and the results route still signs the S3 URL.The
offlinekey inserver/src/config.tsis #2937's definition, copied verbatim, so the two PRs merge cleanly in either order.Knobs
AWS_S3_BUCKET_NAMEtest.env; set inexample.envand CIOFFLINE1/true→ the server streams the export even with a bucketDATA_EXPORT_MAX_CELLS20000000votes.csvis about 2 GB at most, under the 4 GiB zip32 limitTests
conversationZip.test.ts, 20 tests;streamControl.test.ts, 4 tests).finish()reject rather than hang. The abort reaches the producer, and the handler returns when the client goes away.data-export-stream.test.ts, 6 tests, Postgres only)./api/v3/reportExportserves. The file also covers OFFLINE with a bucket, 413 and the non-owner 403.votes.csvquery paused, with no bytes sent for 1.5 s. On disconnect, the handler returns and that backend is gone frompg_stat_activity. With the abort hook removed, this test fails. A reading client gets every row, and nothing is left running.unzip -t, Pythonzipfileandbsdtaraccept the writer's output.tscis clean, ESLint reports 0 errors and Prettier is clean.Docs
docs/export-format.md(from exports and the bulk import declare their vote sign (P-078 PR-E; export bytes unchanged) #2932) gains "The conversation zip": the file list, order, byte-identity, streaming and the 413.docs/configuration.mdgains "Conversation data export": the two modes andDATA_EXPORT_MAX_CELLS.docs/offline.mdlives in offline profile 4: the offline compose overlay, the image bundle script and the install guide (stacked on #2929, #2937, #2936) #2938, which is not in this stack. Adding it here would conflict, so these lines go to offline profile 4: the offline compose overlay, the image bundle script and the install guide (stacked on #2929, #2937, #2936) #2938 instead:/api/v3/dataExport?conversation_id=<id>) downloads a zip of the same files";OFFLINEis set or no S3 bucket is configured (offline PR 5, offline profile 5: the conversation data export streams from the server when no S3 bucket is configured (stacked on #2932) #2940)".Known limits
sendCommentSummaryonly logs a Postgres stream error and never answers, so a failure there leaves the request hanging. This is true of the report export today as well, and this PR does not change that function.No behaviour change when AWS_S3_BUCKET_NAME is set and OFFLINE unset. No schema change.
🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s